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PREFACE 


NATIONAL  AERONAUTICS  AND  SPACE  ADMINISTRATION 
GUIDANCE  FOR  IMPROVING  CUSTOMER  SATISFACTION 


NASA  is  required  to  meet  the  requirements  of  the  President's 
Executive  Order  12862,  Setting  Customer  Service  Standards.  The 
Headquarters  steering  group  is  developing  a  plan  to  meet  those 
requirements.  The  approach  requires  that  each  of  NASA's  Strategic 
Enterprise  Units  develop  its  own  customer  service  plans,  take  a 
satisfaction  survey  of  major  external  customers  (in  the  next  six 
months),  and  target  areas  for  improvement.  A  NASA  agency  progress 
report  is  due  to  the  President  by  8  September  1994. 

The  Logistics  Management  Institute  (LMI)  has  been  engaged  to 

provide  a  common  approach  for  planning,  conducting,  and  analyzing 
customer  satisfaction  surveys.  LMI  has  also  been  asked  to  develop 

a  performance  measurement  framework  for  consistently  undertaking 
and  gauging  improvement  initiatives.  In  this  way,  NASA  will  be  in 
a  position  to  consolidate  individual  customer  service  plans  into  an 
agency  plan. 

LMI  will  be  available  to  the  Strategic  Enterprise  Units  for 

advice  and  guidance  in  accomplishing  these  improvement  initiatives. 
The  authors  can  be  reached  at  telephone  number  (301)  320-2000  for 

such  assistance. 
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EXECUTIVE  SUMMARY 


NATIONAL  AERONAUTICS  AND  SPACE  ADMINISTRATION 
GUIDANCE  FOR  IMPROVING  CUSTOMER  SATISFACTION 


Executive  Order  12862,  Setting  Customer  Service  Standards, 
requires  that  NASA  undertake  customer  satisfaction  surveys  to 
obtain  customer  feedback  for  improving  its  products  and  services. 
It  also  requires  NASA  to  search  for  best  industry  methods  and 
incorporate  them  into  current  practices  as  appropriate.  Executive 
Order  12862,  however,  does  not  provide  much  in  the  way  of 
guidelines  for  its  implementation. 

This  paper  provides  a  comprehensive  set  of  guidelines  for 
complying  with  the  provisions  of  Executive  Order  12862.  The 
guidance  is  presented  as  follows: 

Section  1:  Introduces  the  requirements  of  Executive  Order 
12862  and  shows  the  relationship  of  those  requirements  to  other 
NASA  improvement  efforts. 


Section  2:  Outlines  NASA's  Strategic  Enterprise  Units  and 
their  customers.  It  also  defines  an  appropriate  level  of  process 
aggregation  for  the  Strategic  Enterprise  Units  to  design  customer 
satisfaction  surveys  that  can  directly  lead  to  performance 
improvements  for  NASA. 


Section  3:  Delineates  a  quality  measurement  framework  to  guide 
the  focus  of  customer  satisfaction  surveys.  It  also  provides 
practical  advice  on  planning,  conducting,  and  analyzing  the 
customer  surveys  as  well  as  a  basis  for  prioritizing  performance 
improvement  initiatives. 


Section  4:  Offers  a  total  performance  measurement  framework-- 
combining  cost,  output,  and  quality — to  further  guide  the 
accomplishment  of  industry  benchmarking,  continual  improvement,  and 
implementation  of  the  Performance  and  Results  Act  of  1993.  It  uses 
a  NASA  case  study  to  demonstrate  the  utility  of  this  performance 
measurement  framework . 
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NATIONAL  AERONAUTICS  AND  SPACE  ADMINISTRATION 
GUIDANCE  FOR  IMPROVING  CUSTOMER  SATISFACTION 


1 .  INTRODUCTION 

IMPROVEMENT  EFFORTS 

During  the  spring  of  1992,  the  National  Aeronautics  and  Space 
Administration  (NASA)  embarked  on  a  continual  improvement  program 
to  enhance  customer  satisfaction  with  its  products  and  services. 
NASA's  continual  improvement  effort,  which  focuses  both  on  reducing 
costs  and  increasing  quality,  uses  a  traditional  five-element 
improvement  cycle.  Figure  1-1  illustrates  NASA's  improvement  cycle. 


FIGURE  1-1.  NASA's  Improvement  Cycle 
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The  five  elements  of  NASA's  improvement  cycle  are: 

(1)  Customers  judge  the  results  (through  the  use  of  advisory 
groups  and  formal  surveys) 

(2)  Process  definition  provides  the  understanding  for 
addressing  customer  concerns 

(3)  Management  and  employee  groups  (the  stakeholders)  work  in 
teams  to  improve  processes 

(4)  Leadership  gives  focus,  alignment,  and  direction  to  team 
efforts 

(5)  Tools  and  techniques  are  used  to  bring  about  process 
improvement;  they  include  cross-functional  problem  solving, 
controlled  testing,  and  performance  measurement. 

Like  other  Government  agencies,  NASA  is  required  to  pursue  two 
related  improvement  initiatives.  First,  in  September  1993,  the 
President  issued  Executive  Order  12862,  Setting  Customer  Service 
Standards,  to  help  implement  the  recommendations  of  the  National 
Performance  Review.  Executive  Order  12862,  as  with  NASA's 
continual  improvement  effort,  focuses  on  reducing  costs  and  in¬ 
creasing  product/ service  quality  to  improve  customer  satisfaction. 
However,  the  Executive  order  places  special  emphasis  on  taking 
formal  surveys  to  obtain  customer  feedback  and  on  benchmarking  re¬ 
lated  industry  practices  to  improve  Government  processes.  Govern¬ 
ment  agencies  are  to  begin  those  activities  in  FY94 . 

Second,  the  Performance  and  Results  Act  of  1993  requires  NASA 
not  only  to  improve  costs  and  enhance  quality,  but  also  to  focus  on 
outputs  and  performance  budgeting.  It  further  requires  pilot  tests 
to  develop  strategic  plans,  to  construct  operating  plans,  and  to 
formulate  performance  budgets  (in  terms  of  outputs,  quality,  and 
costs)  .  The  pilot  tests  are  to  be  completed  by  September  1997,  with 
a  full  performance  budgeting  system  in  operation  by  the  year  2000. 

We  believe  that  all  of  these  efforts — continual  improvement. 
Executive  Order  12862,  and  the  Performance  and  Results  Act  of  1993— 
should  be  thought  of  as  integral  to  an  overall  performance 
improvement  system.  Certainly,  the  five-stage  continual  improvement 
cycle  is  needed  to  make  the  other  two  efforts  effective.  Also,  a 
cohesive  performance  measurement  system  provides  the  capability  to 
focus  on  all  aspects  of  performance  (cost,  quality,  and  output) , 
coordinate  improvement  efforts,  and  gauge  the  results. 

EXECUTIVE  ORDER  12862 

Executive  Order  12862  requires  that  NASA  take  customer 
satisfaction  surveys  to  obtain  customer  feedback  for  improving  its 
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products  and  services.  It  also  requires  NASA  to  search  for  best 

industry  methods  and  incorporate  them  into  current  practices  as 

appropriate.  The  Executive  order,  however,  does  not  provide  much  in 
the  way  of  guidelines  for  its  implementation. 

NASA  needs  a  comprehensive  set  of  guidelines  for  complying 
with  the  general  provisions  of  Executive  Order  12862.  This  guidance 
answers  the  following  questions: 

(a)  Who  are  NASA's  general  customers? 

(b)  Who  in  the  NASA  organization  should  be  taking  customer 

satisfaction  surveys  and  searching  for  best  industry  models? 

(c)  What  should  NASA  measure  to  obtain  meaningful  customer 

feedback  and  relevant  industry  information  for  improving 
performance? 

(d)  How  should  NASA  approach  customer  satisfaction  surveys? 

(e)  When  should  NASA  undertake  customer  satisfaction  surveys 
and  search  for  best  industry  practices? 


ORGANIZATION  OF  PAPER 

This  guidance  presents  a  strategy  for  NASA  to  meet  the 
requirements  of  Executive  Order  12862,  to  further  its  continual 
improvement  efforts,  and  to  prepare  for  the  challenges  of  the 
Performance  and  Results  Act  of  1993. 

Section  2  discusses  NASA's  customers  and  organizational 
structure,  and  the  concept  of  a  process  for  diagnosing  problems  and 
making  process  improvements . 

Section  3  focuses  on  meeting  the  requirements  of  Executive 
Order  12862.  It  defines  a  measurement  framework  for  assessing 
customer  satisfaction  and  outlines  the  basic  steps  for  planning, 
conducting,  and  analyzing  customer  satisfaction  surveys. 

Section  4  presents  a  more  complete  performance  measurement 
system.  It  contains  the  measurement  framework  for  assessing 
customer  satisfaction  discussed  in  the  previous  section,  and  other 
performance  measures  that  are  needed  to  meet  the  industry 
benchmarking  requirements  of  Executive  Order  12862,  continual 
improvement  efforts,  and  the  Performance  and  Results  Act  of  1993. 
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2.  DEFINITION  OF  A  PROCESS 


Everything  NASA  does  involves  a  process .  Every  process 
includes  one  or  more  inputs  (facilities,  management,  and  research 
staff);  a  variety  of  transformations;  and  various  outputs  and 
levels  of  quality.  Figure  2-1  illustrates  the  most  simple  process, 
with  customer  and  supplier  feedback. 


FIGURE  2-1.  A  SIMPLE  PROCESS 


CUSTOMERS 

NASA  is  currently  undertaking  an  input/output  analysis  to 
fully  identify  its  customers,  products  and  services,  inputs, 
suppliers,  and  associated  processes.  In  broad  terms,  that  analysis 
has  shown  that  NASA's  external  customers  are  the  science  and 
education  communities,  the  aerospace  industry,  the  aeronautics  and 
aviation  industries,  and  other  government  agencies.  Five  Strategic 
Enterprise  Units  (SEUs)  have  been  defined  to  focus  on  strategic 
objectives  and  external  customers. 

NASA's  five  SEUs  are  Mission  to  Planet  Earth,  Aeronautics, 
Human  Exploration  and  Development  of  Space,  Scientific  Research, 
and  Space  Technology.  The  SEUs  differ  in  their  customer  focus.  For 
example,  the  Aeronautics  Enterprise  serves  the  aeronautics  industry 
and  other  U.S.  government  agencies,  while  the  Space  Technology 
Enterprise  serves  the  aerospace  industry.  However,  some  SEUs  serve 
some  of  the  same  external  customers.  For  example,  although  many  of 
the  external  customers  of  the  Mission  to  Planet  Earth  Enterprise 
and  the  Scientific  Research  Enterprise  differ  substantially,  they 
both  serve  the  science  community  in  various  ways.  Also,  some  of 
the  customers  are  general  rather  than  specific,  such  as  customers 
of  public  affairs  or  publications. 

NASA  headquarters  has  indicated  that  each  SEU  will  be 
responsible  for  developing  a  customer  service  plan  to  meet  the 
requirements  of  Executive  Order  12862.  Each  SEU  should  plan  and 
conduct  at  least  one  customer  survey  during  the  next  six  months.  We 
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now  address  the  appropriate  process  aggregation  level  for  SEUs  to 
undertake  customer  satisfaction  surveys  and  other  performance 
initiatives . 

PROVIDERS 

NASA  needs  to  define  the  appropriate  organizational  level  in 
its  organization  for  defining  its  processes  and  for  meeting  the 
requirements  of  Executive  Order  12862  and  other  performance 
improvement  initiatives.  We  believe  that  NASA  should  not  define 
its  processes  too  broadly  or  too  narrowly  to  effectively  undertake 
performance  improvement. 

In  principle,  we  should  consider  at  least  four  levels  of 
aggregation  for  defining  NASA's  processes.  First,  NASA  could 
consider  each  of  its  five  SEUs  as  a  separate  process.  Second,  at  a 
somewhat,  more  detailed  level,  NASA  could  consider  each  of  its  12 
current  budget  programs  (e.g..  Launch  Services,  or  Physics  and 
Astronomy)  as  individual  processes.  Third,  at  a  still  more 
disaggregated  level,  NASA  could  define  separate  processes  for  the 
specific  products /services  it  provides  at  each  of  its  nine  research 
centers  or  at  its  Headquarters.  (As  an  example,  one  process  could 
consist  of  all  wind  tunnels  at  Ames  Research  Center) .  Fourth,  at 
an  even  more  detailed  level,  NASA  could  define  separate  processes 
for  each  of  its  functional  areas,  such  as  maintenance  and  repair  of 
buildings,  facilities,  and  structures. 

NASA  needs  some  criteria  to  select  the  most  appropriate 
definition  of  process  aggregation  for  undertaking  performance 
improvement  initiatives.  We  believe  that  four  criteria  have 
application  to  NASA.  A  process  should  do  the  following: 

(1)  Encompass  products/services  that  are  produced  jointly 
through  the  use  of  common  staff,  equipment,  facilities,  and 
technology.  In  economics,  this  criterion  would  be  referred  to  as 
the  production  function. 

(2)  Come  under  the  control  of  a  manager  who  is  directly 
responsible  and  accountable  for  strengthening  operations  and  im¬ 
proving  customer  satisfaction  related  to  certain  products /services. 
For  NASA,  this  criterion  means  that  each  process  needs  to  be 
defined  in  terms  of  the  relevant  research  center  and  headquarters 
SEU  that  oversees  its  operation. 

(3)  Serve  as  a  diagnostic  tool  for  taking  action  if  specific 
performance  improvements  are  warranted  by  customer  feedback.  This 
criterion  means  that  the  process  is  keyed  to  the  products/services 
provided  to  specific  customers,  and  the  process  is  detailed  enough 
to  pinpoint  actionable  items. 
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(4)  Point  to  areas  in  which  industry  best  practices  might  be 
useful  for  making  improvements.  NASA  should  be  able  to  identify 
specific  functions  (or  a  group  of  functions)  from  its  processes, 
search  for  industry  methods  that  relate  to  those  functions,  and 
incorporate  the  best  industry  practices  into  its  processes.  In  the 
continual  improvement  literature,  this  practice  is  referred  to  as 
"benchmarking. m1 

We  now  apply  these  criteria  to  select  the  most  appropriate 
level  of  aggregation  for  defining  NASA  processes. 

Table  2-1  shows  a  matrix  of  the  four  alternative  process 
aggregation  levels  (the  rows)  and  the  four  selection  criteria  (the 
columns)  .  We  indicate  a  "yes"  in  the  cells  when  the  process 
aggregation  levels  satisfy  the  specific  criteria  for  process 
aggregation  and  "no"  when  they  do  not. 

At  one  extreme,  the  matrix  shows  that  defining  processes  at 
the  strategic  enterprise  unit  level  is  too  broad  for  undertaking 
performance  improvements.  At  the  other  extreme,  defining  processes 
at  the  functional  level  is  too  narrow  to  diagnose  all  potential 
customer  concerns  in  a  specific  product  or  service.  As  Table  2-1 
shows,  all  four  criteria  are  satisfied  only  for  the 
products /services  at  each  of  NASA's  research  centers  or  at 
headquarters.  It  is  at  this  level  of  process  aggregation  that 
performance  improvement  initiatives  align  with  the  managers  who 
directly  diagnose  performance  problems,  consider  the  best  practices 
of  industry,  and  make  process  improvements. 

We  conclude  that  NASA  should  define  its  processes  along 
product  or  service  lines  at  each  of  the  nine  research  centers  or  at 
headquarters.  For  example,  the  Ames,  Lewis,  and  Langley  Research 
Centers  all  operate  and  manage  wind  tunnels  for  both  NASA  and 
industry  customers.  Nonetheless,  a  separate  wind-tunnel  process 
should  be  defined  for  each  of  the  three  research  centers.  If  this 
is  done,  the  three  separate  wind-tunnel  processes  would  reflect  the 
differences  in  their  operations,  thereby  fostering  customer 
feedback,  industry  benchmarking,  and  improvement  initiatives.  The 
Aeronautics  Enterprise  would  aggregate  and  report  on  wind-tunnel 
processes  and  other  processes  that  fall  under  its  purview. 


1See  Robert  C.  Camp,  Benchmarking:  The  Search  for  Industry- 
Best  Practices  that  Lead  to  Superior  Performance,  White  Plains 
New  York:  Quality  Resources,  1989,  for  an  in-depth  treatment  of 
benchmarking . 
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Table  2-1 


SELECTION  OF  PROCESS  DEFINITION 


Process  aggregation 
Level 

Criteria  for  selection 

Production 

function 

Direct  manager 
control 

Diagnostics  for  improving 
customer  satisfaction 

Industry 

benchmarking 

Strategic  enterprise 
unit 

Yes 

No 

No 

No 

Budget  program 

Yes 

No 

No 

No 

Products/services  at 
research  centers  or 
headquarters 

Yes 

Yes 

Yes 

Yes 

Function 

Yes 

Yes 

No 

Yes 
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3.  EXECUTIVE  ORDER  128G2 


Executive  Order  12862  requires  NASA  to  develop  customer 
satisfaction  surveys  and  performance  industry  benchmarking  in  order 
to  improve  its  operations.  We  believe  that  NASA  should  first  focus 

on  taking  customer  satisfaction  surveys  (for  at  least  a  year) 

before  undertaking  industry  benchmarking.  This  will  give  NASA  an 
opportunity  to  understand  its  customer  standards  well,  a  necessary 
first  step  before  considering  the  introduction  of  best  industry 
standards . 

Also,  industry  benchmarking  requires  not  only  a  focus  on 

quality — as  is  the  case  for  customer  satisfaction  surveys — but  also 
on  cost  and  output  performance.  NASA  will  have  to  develop  cost  and 
output  measures  before  it  can  properly  undertake  industry 
benchmarking.  Section  4  provides  the  broader  measurement  system 
that  NASA  will  need  to  comply  with  the  industry  benchmarking 

requirements  of  the  Executive  Order  (and  with  continual  improvement 
and  the  Performance  and  Results  Act) . 

Presented  below  is  the  quality  measurement  approach  needed  for 
meeting  the  customer  survey  requirements  of  the  Executive  order. 

It  is  followed  by  guidance  for  planning,  conducting,  and  analyzing 
customer  surveys. 

QUALITY  MEASUREMENT 

Executive  Order  12862  basically  requires  that  the  customer 
survey  effort  provide  data  for  determining  status  of  quality  in 
NASA  products/services,  for  prioritizing  NASA  improvement  initia¬ 
tives,  and  for  gauging  overall  progress. 

The  Executive  order  requires  that  the  customer  survey 
instrument  cover  the  following  four  areas: 

(a)  Customer  standards  for  each  quality  dimension  of  a 
product/service 

(b)  Customer  satisfaction  levels  for  every  product /service 
quality  dimension 

(c)  Customer  importance  attached  to  each  product /service 
quality  dimension 

(d)  Customer  concern  with  product /service  expenditures 

Surveys  need  to  be  structured  so  that  the  respondents  register 
their  feelings  in  quantitative  terms.  Such  scoring  should  be  done 
for  each  of  customer  standards,  satisfaction  levels,  importance 
levels,  and  expenditure  concerns.  The  resultant  data  then  can  be 
combined  to  meet  Executive  Order  12862  requirements. 
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Equation  3-1  shows  how  to  combine  much  of  the  data  into  a 
quality  index.  In  accordance  with  our  definition  of  process,  the 
quality  index  represents  "n"  quality  factors  of  a  product/service 
that  is  produced  by  a  research  center  or  by  headquarters  for  a 
certain  set  of  customers.  Scores  on  actual  and  standard  levels  are 
compared  and  then  weighted  by  relative  customer  importance  scores. 
(These  scores  are  obtained  through  the  customer  survey  as  explained 
later  in  this  section.) 

QI=w(l)x [actual  quality (1) /standard  quality (1) ] +  [Eq.  3-1] 

w (2 )x [actual  quality (2) /standard  quality (2) ]+.. .+ 
w(n)x  [actual  quality (n) /standard  quality (n) ] 

Equation  3-1  is  a  useful  tool  for  determining  the  quality 
factors  most  in  need  of  improvement.  As  an  example,  if  two  quality 
factors  achieve  only  40  percent  of  their  standard  levels,  then  the 
one  with  the  greater  customer  weight  should  receive  the  highest 
priority  for  making  improvements.  In  contrast,  a  quality  factor 
with  an  80  percent  achievement  level  relative  to  its  standard  (and 
a  relatively  low  customer  weight)  should  receive  a  lower  priority 
for  implementing  improvements.  Moreover,  these  quality  shortfalls 
also  should  be  compared  with  customer  expenditure  concerns  for  the 
product /service.  It  is  possible  that  customer  expenditure  concerns 
are  as  great  as  or  greater  than  any  particular  quality  shortfall, 
and  that  should  be  taken  into  account  before  prioritizing  NASA 
improvement  efforts.  See  Section  4  for  a  more  complete  discussion 
of  customer  expenditure  concerns  and  NASA  costs  for  producing 
products/services . 

The  quality  index  is  also  a  useful  tool  for  measuring  the 
degree  to  which  overall  quality  standards  are  met.  For  example,  a 
70  percent  quality  index  score  would  mean  that,  on  the  average,  the 
process  is  achieving  70  percent  of  the  quality  standards  desired  by 
the  customers.  The  maximum  quality  index  score  is  100  percent  and 
the  minimum  is  0  percent. 

CUSTOMER  SURVEY 

There  are  three  phases  for  undertaking  customer  surveys: 
planning  for  the  survey,  conducting  the  survey,  and  analyzing  its 
results . 

Survey  Plan 

The  design  of  a  survey  includes  identifying  NASA's  customers, 
structuring  the  questionnaire,  and  planning  for  sampling  the 
customers.  All  of  these  considerations  are  important  for  ensuring 
that  NASA  obtains  accurate  and  unbiased  feedback  from  its 
customers. 
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NASA  is  already  defining  its  customer  base  in  precise  terms. 
This  is  an  important  first  step  toward  meeting  the  requirements  of 
Executive  Order  12862. 

The  customer  questionnaire  should  be  developed  using  a  working 
group  for  each  process.  For  example,  a  working  group  could  be 
formed  for  the  wind-tunnel  process  at  the  Ames  Research  Center. 
This  group  could  include  the  manager  of  the  process,  selected 
members  of  the  manager's  staff,  a  key  customer,  and  a  survey 
expert.  The  manager  and  staff  would  ensure  that  customer  feedback 
is  meaningful  for  making  improvements  to  their  process.  The 
customer  participant  would  increase  the  likelihood  that  important 
product /service  characteristics  are  included  in  the  questionnaire, 
while  the  survey  expert  would  help  the  group  to  avoid  unclear  and 
biased  wording  in  the  questionnaire. 

Survey  research  has  shown  that  a  mail  questionnaire  needs  to 
follow  a  certain  structure  to  obtain  an  acceptable  response  rate. 
First,  it  should  include  a  cover  letter  describing  the  survey’s 
importance.  The  cover  letter  should  include  the  following:  NASA 
intends  to  take  periodic  surveys  to  determine  how  well  it  is 
meeting  the  needs  of  its  customers;  customer  feedback  will  help 
NASA  make  improvements  to  its  processes;  customer  cooperation  is 
essential  for  continually  improving  NASA's  products /services;  and 
NASA  guarantees  that  individual  respondent  data  will  be  kept 
confidential . 

Second,  the  survey  questions  should  be  written  in  sections: 
one  section  on  customer  satisfaction  levels,  another  on  customer 
standards,  still  another  on  customer  expenditure  concerns,  and 
finally  one  on  the  importance  that  customers  attach  to  specific 
product/service  characteristics.  The  questions  should  be  structured 
so  that  respondents  can  express  their  feelings  in  degrees  of 
intensity.  Survey  research  further  shows  that  use  of  a  7-point 
scale,  with  appropriate  descriptive  labels,  is  very  effective. 
Using  the  example  of  the  Ames  Research  wind  tunnel,  a  survey 
question  on  customer  satisfaction  levels  could  be  as  follows: 


How  satisfied  are  you  with  the  reliability  of  wind-tunnel 
services  provided  by  the  Ames  Research  Center: 


1- Not  satisfied? 

2- Slightly  satisfied? 

3 - Somewhat  satisfied? 

4- Fairly  satisfied? 

5- Quite  satisfied? 

6- Very  satisfied? 

7- Extremely  satisfied? 
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All  sections  should  conclude  with  a  question  that  elicits 
customer  explanations  for  relatively  negative  answers  (such  as  1,2, 
or  3  in  the  above  sample  question)  or  general  comments. 

After  the  questionnaire  sections,  the  survey  should  conclude 
with  two  additional  types  of  questions.  One  should  address  the 
customers'  overall  rating  of  NASA's  products /services,  again  using 
a  l-to-7  scoring  scale.  Such  a  question  is  useful  for  checking  the 
consistency  of  customer  responses  throughout  the  questionnaire.  The 
other  question  should  be  aimed  at  developing  a  business  profile  of 
the  customer.  That  information  should  be  useful  in  determining 
whether  different  types  of  customers  have  similar  feelings  about 
various  aspects  of  NASA's  products /services,  which  may  help  NASA 
target  its  products/services  to  specific  customers. 

Before  conducting  the  survey,  each  working  group  should  pre¬ 
test  the  questionnaire  to  check  for  possible  bias  and  ensure  that 
the  questions  are  clear  and  effective.  The  pretesting  should  result 
in  the  elimination  of  long,  awkwardly  worded,  or  ambiguous 
questions;  redundant  questions;  and  wording  that  may  offend  or 
sound  foolish  to  respondents .  Questionnaires  are  routinely  improved 
when  pretested. 

A  mail  survey,  by  itself,  is  often  not  sufficient  for 
obtaining  an  adequate  survey  response  rate.  The  Office  of  Manage¬ 
ment  and  Budget  requires  that  Government  customer  satisfaction 
surveys  strive  for  a  75  percent  response  rate.  Survey  research 
shows  that  a  mail  survey  often  achieves  only  a  40  percent  response 
rate.  To  increase  that  response  rate,  organizations  need  to  take 
two  additional  actions — a  reminder  card  and  a  follow-up  with  a 
telephone  call  or  in-person  interview.  These  steps  —  a  mail  survey, 
followed  by  a  card  and  telephone/ in-person  interview  —  are  referred 
to  as  the  "total  design  method"  in  survey  research  circles. 

Table  3-1  compares  the  various  survey  methods  and  illustrates 
that  the  total  design  method  often  achieves  acceptable  response 
rates  at  reasonable  costs.2  A  questionnaire  generally  costs  about 
$5.00  for  handling  and  mailing.  However,  because  only  40  percent 
are  returned,  the  cost  of  the  initial  and  follow-up  mail 
questionnaires  increases  to  approximately  $12.00  ($5/0.40)  in  order 

to  obtain  satisfactory  response  rates.  Telephone  and  in-person 
interviews  are  more  expensive  than  mail  questionnaires,  but  their 
response  rates  are  considerably  higher  than  those  for  mail 
questionnaires.  The  total  design  method  combines  the  best  of  all 
survey  methods  by  balancing  costs  with  response  rate 
considerations . 


2See  D.A.  Di liman.  Mail  and  Telephone  Surveys:  The  Total 
Design  Method,  New  York:  John  Wiley  &  Sons,  1978,  for  an  in-depth 
treatment  of  this  survey  method. 
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TABLE  3-1 

SURVEY  METHODS  AND  COSTS 


Survey  method 

Response  rate 
% 

Average  cost  $ 

Per-attempted  contact 

Per-successful 

contact 

1.  Mail  questionnaire 

40 

5 

12 

2.  Telephone  interview 

60 

20 

33 

3.  In-person  interview 

70 

40 

57 

4.  Total  design  method 

65 

17 

26 

Source:  Various  survey  research  documents. 


Notes:  Mail  questionnaire  costs  include  clerical  and  data  coding 
functions.  In-person  interviews  do  not  include  travel /lodging 
costs.  Total  design  method  assumes  a  mixture  of  mail 
questionnaires,  telephone  calls,  and  in-person  interviews.  If  NASA 
has  additional  internal  costs,  they  should  be  calculated  and 
considered. 

The  sample  size  of  the  survey  affects  both  its  accuracy  and 
cost.  Since  many  of  NASA  processes  involve  fewer  than  250  industry 
and  internal  NASA  customers,  we  suggest  that  NASA  survey  all  of  its 
customers.  However,  when  the  customer  base  exceeds  250  customers, 
NASA  should  take  a  50  percent  sample.  Such  sampling  rate  procedures 
balance  data  accuracy  considerations  against  survey  costs.3 

Survey  Conduct 

NASA  should  take  a  survey  biannually  to  periodically  review 
performance  progress  and  continuously  obtain  customer  feedback.  It 
should  also  use  a  100  percent  sampling  rate  when  the  customer  base 
is  under  250  and  50  percent  when  more  than  250. 

NASA  should  pre-test  every  survey  instrument.  We  recommend 
that  the  working  group  test  the  survey  instrument  by  a  combination 
of  telephone  call  and  in-person  interviews.  A  five  percent  pre-test 
sample  should  be  sufficient.  A  useful  procedure  for  pretesting  is 


3A  random  sample  may  be  taken  when  the  customer  base  is 
relatively  homogenous.  However,  when  the  customer  base  is 
diverse,  the  method  of  stratified  sampling  is  more  appropriate 
for  obtaining  the  desired  data  accuracy.  See  George  W.  Snedecor 
and  G.  Cochran,  Statistical  Methods,  The  Iowa  State  University 
Press,  1989,  for  a  practical  discussion  of  sampling  techniques. 
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to  ask  respondents  a  question  such  as,  "Why  did  you  give  such  a 
rating  in  answer  to  this  question?"  The  answers  may  indicate  that 
the  respondents  are  interpreting  the  questions  differently  from  the 
way  the  working  group  had  intended. 

After  the  pre-test,  an  additional  6— week  period  should  he  set 
aside  for  collecting  questionnaire  responses.  The  questionnaires 
should  be  mailed  out  with  a  stamped,  self-addressed  return 
envelope.  After  the  second  week,  NASA  should  mail  a  card  thanking 
respondents  who  have  completed  the  questionnaire  and  reminding 
others  that  it  is  looking  forward  to  receiving  their  responses. 
After  the  fourth  week,  NASA  should  make  any  follow-up  telephone 
calls  and  personal  interviews  as  is  necessary  to  achieve  high 
response  rates . 

Expert  advice  is  especially  important  in  conducting  the 
survey.  The  results  will  be  accurate  only  if  based  upon  a 
reasonable  cross-section  of  the  population.  Further,  customer 
interviews — either  telephone  or  face-to-face  could  introduce  bias 
into  the  survey  data  if  not  properly  conducted.  NASA  may  want  to 
contract  for  those  services . 

Survey  Analysis 

Before  analyzing  the  data,  NASA  should  code  the  data  for 
entering  into  a  computer.4  It  should  train  specific  sections  of  the 
organization  to  transfer  the  survey  data  into  the  computer.  Entries 
should  be  double-checked  to  ensure  accuracy. 

The  amount  of  information  yielded  by  customer  surveys  can  be 
considerable.  In  general,  analysis  of  survey  data  should  include 
examination  of  the  responses  to  every  question  for  each  category  of 
respondent.  Such  information  indicates  which  products /services  are 
meeting  customer  standards  for  different  categories  of  NASA 
customer.  Also,  examination  of  the  responses  to  open-ended  comment 
questions,  such  as  why  the  respondents  gave  negative  answers  to 
some  questions,  may  suggest  process  improvements.  Finally,  more 
sophisticated  statistical  approaches  may  be  used  for  determining 
the  relationship  between  the  responses  to  various  questions, 
thereby  improving  understanding  of  customer  thinking. 


4The  survey  analysis  should  be  conducted  with  a  common 
statistical  package  to  facilitate  analytical  work  and  com¬ 
parisons.  We  have  designated  the  Statistical  Analysis  System 
(SAS)  for  this  purpose. 
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4.  PERFORMANCE  MEASUREMENT  SYSTEM 

In  this  section,  we  define  a  performance  measurement  system 
that  addresses  all  of  NASA's  immediate  and  long-term  performance 
initiatives— Executive  Order  12862,  continual  improvement,  and  the 
Performance  and  Results  Act  of  1993.  We  then  demonstrate  that 
system  by  applying  it  to  a  wind-tunnel  case  study. 

MEASUREMENT  FRAMEWORK 

A  performance  measurement  system  must  provide  comprehensive 
signals  for  an  organization  to  improve  performance.  A  comprehensive 
performance  measurement  system  has  three  essential  components:  the 
ability  to  enhance  quality,  to  raise  output  capability,  and  to 
lower  costs.  Failure  to  address  any  of  these  concerns  will  always 

result  in  performance  shortfalls.  If  an  organization  focuses 
largely  on  quality,  it  may  very  well  enhance  quality  but  its  output 
and  costs  could  suffer.  (Mercedes  Benz  could  be  a  good  example  of 
an  organization  that  focused  too  much  on  quality  and  not  enough  on 
other  factors).  Alternatively,  if  an  organization .  focuses  mostly 
on  output,  it  may  very  well  raise  output  capability,  but  its 

quality  and  costs  could  suffer.  (The  American  car  industry  in  the 

early  1980s  may  be  a  good  example  of  this  type  of  focus.)  Finally, 

if  an  organization  focuses  largely  on  reducing  costs,  it  may  very 
well  become  more  efficient  but  its  output  and  quality  might  suffer 
(e.g.,  the  Yugo) .  Only  when  an  organization  focuses  simultaneously 
on  quality,  output,  and  cost  can  it  truly  establish  a  high  level  of 
overall  performance  and  customer  satisfaction. 

We  use  the  concept  of  performance  unit  cost  to  ensure  that 
organizations  focus  on  all  aspects  of  performance  simultaneously. 
Performance  unit  cost  is  defined  as  costs  relative  to  outputs  at 
defined  quality  levels.  Figure  4-1  illustrates  the  process  for 
integrating  costs,  output,  and  quality. 

Performance  unit  cost  incorporates  the  best  features  of 
Federal,  state,  and  local  government  budgeting  efforts.  Measures 
of  efficiency  (cost  per  unit  of  output)  have  been  used  extensively 
in  the  Federal  government,  especially  within  the  Department  of 
Defense.  At  state  and  local  governments,  measures  of  effectiveness 
or  outcome  (output  adjusted  for  quality  differences)  also  have  been 
used  extensively.  More  recently,  state  and  local  governments  have 
combined  effectiveness  and  efficiency  measures  into  performance 
unit  cost  or  cost  per  unit  of  outcome.  The  Governmental  Accounting 
Standards  Board  recently  has  endorsed  this  approach.5 


5  Proposed  Statement  of  the  Government  Accounting  Standards 
Board  on  Concepts  Related  to  Service  Efforts  and  Accomplishing 
Reporting,  Governmental  Accounting  Standards  Board,  September  15, 
1993. 
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FIGURE  4-1.  MEASUREMENT  INTEGRATION  PROCESS 

We  now  define  the  elements  of  performance  unit  cost.  First, 
we  discuss  the  concepts  of  cost,  output,  and  quality.  T  en  we 
discuss  the  integration  of  those  concepts  into  a  performance  unit 
and  performance  unit  cost. 


Cost 

Ideally,  process  costs  should  be  inclusive  _  of  operating 
outlays;  appropriate  allocations  of  general,  administrative,  and 
other  fixed  costs;  and  depreciation  charges  for  use  of  capital 
items  (such  as  buildings,  facilities,  and  structures).  However, 
many  government  cost  accounting  systems  cannot  support  this  concept 
of  costs  without  substantial  upgrades.  Some  of  the  problems  with 
cost  accounting  systems  include  the  following:  they  cannot  readily 
track  variable  costs  to  specific  processes,  they  do  not  include 
procedures  for  depreciating  capital  items,  and  they  _  lack  the 
capability  to  allocate  fixed  costs  to  processes.  Recognizing  such 
potential  difficulties,  the  Performance  and  Results  Act  of  1993 
does  not  require  full  implementation  of  sophisticated  cost 
accounting  and  performance  budgeting  until  the  year  2000.  Also, 
industry  benchmarking  and  continual  improvement  require  such  cost 
accounting  systems  to  be  most  effective.  NASA  should  ensure  that 
its  cost  accounting  and  budgeting  systems  are  sufficient  for 
meeting  these  performance  initiatives. 
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Output 

Some  NASA  processes  may  produce  homogeneous  products /services, 
but  most  are  likely  to  produce  diverse  products /services.  The 
measurement  of  homogeneous  outputs  is  straightforward--simply  add 
the  number  of  units  produced  (e.g.,  service  hours)  during  a  given 
time  period.  But,  the  measurement  of  diverse  pro-ducts /services  is 
more  difficult  and  requires  the  use  of  indexes. 

Table  4-1  illustrates  the  construction  of  output  indices  when 
the  process  yields  three  different  kinds  of  services.  Suppose  the 
different  services  are  S(l),  S(2),  and  S(3).  Each  could  be  measured 

in  terms  of  hours  used,  but,  because  they  contribute  differently  to 
the  total  process  output,  they  must  be  weighted  to.  take  account  ,  of 
the  differences.  The  weights,  denoted  by  w(i),  might  be  relative 
expenditures  or  revenues  and  are  fixed  over  time  and  . sum.  to  unity. 
The  contribution  of  each  service  to  the  total  output  is  -its  service 
level  multiplied  by  its  respective  weight  [e.g.,S(l)x  w.(l)],  and 
the  summation  of  all  service  contributions  provides  an  indication 
of  total  output. 


Table  4-1 

CONSTRUCTING  OUTPUT  INDICES 


Output  service 
(S) 

Importance  weight 
(w) 

Index  contribution 
(Sxw) 

S(l) 

w(l) 

S(l)xw(l) 

S(2) 

w(2) 

S(2)xw(2) 

S(3) 

w(3) 

S(3)xw(3) 

Total 

1.0 

Index  value 

Quality 

We  now  need  to  define  quality  factors  for  NASA's  products/ 
services  and  to  express  NASA's  outputs  in  quality-equivalent  .  terms . 
NASA's  products  and  services  normally  are  expected  to  involve 
multiple  quality  factors  —  such  as  suitability,  reliability,  and 
usability  —  so  the  same  basic  index  approach  used  to  summarize 
outputs  also  has  application  to  the  summation  of  quality.  However, 
two  major  differences  in  constructing  quality  indexes  exist. 

First,  unlike  the  output  index,  the  weights  are  derived  from 
customer  satisfaction  surveys  (as  required  by  Executive  Order 
12862)  and  not  from  NASA's  revenue  or  expenditure  data.  In  the 
survey,  customers  would  rate  the  importance  of  each  quality,  factor, 
and  those  ratings  would  be  used  to  construct  the  quality  weights  in 
the  index. 
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Second,  the  performance  of  quality  factor  is  measured  in 
percentage  terms  -  rather  than  in  actual  values  as  in  the  case  of 
output  measurement  -  to  determine  how  the  actual  quality  levels 
compare  to  customer  standards.  Moreover,  industry  benchmarking 
could  also  be  used  as  standards  in  the  quality  index,  but  they 
would  have  to  be  as  least  as  high  as  the  customer's  standards.  We 
believe,  however,  that  benchmarking  should  be  used  only  after  using 
customer  standards  for  a  period  of  time. 

Equation  4-1  shows  the  general  form  of  the  quality  index  with 
"n"  quality  factors: 

+  [Eq.4-1] 


QI  =w(l)x  [actual  quality (1)  /standard  quality  (1)] 
w(2)x  [actual  quality (2) /standard  quality  (2)] 


+  *  •  .  + 


w 


(n)x  [actual  quality (n) /standard  quality  (n) ] 


The  quality  index  then  measures  the  degree  to  which _  the 
quality  standards  are  met.  For  example,  an  80  percent  quality_  index 
score  would  mean  that,  on  the  average,  the  process  is  achieving  80 
percent  of  the  quality  standards  desired  by  the  customers.  _  The 
maximum  quality  index  score  is  100  percent  and  the  minimum  is  0 
percent. 

Equation  4-1  is  also  a  useful  tool  for  determining  the  quality 
factors  that  have  the  greatest  need  for  improvement.  If  two  quality 
factors  achieve  only  50  percent  of  their  standard  levels,  for 
example,  then  the  one  with  the  greater  customer  weight  shou 
receive  the  higher  priority  for  making  improvements.  In  contrast, 
a  quality  factor  with  a  90  percent  achievement  level  relative  to 
its  standard  (and  a  relatively  low  customer  weight)  should  receive 
a  lower  priority  for  implementing  improvements. 

Performance  Unit 

Performance  units  are  outputs  standardized  at  perfect  quality 
levels.  Such  standardization  is  needed  to  measure  performance 

accurately  and  to  encourage  high-quality  performance.  Table  4  2 
illustrates  three  different  cases.  Case  -a"  shows  that  1,200  units 
of  output  were  produced  at  an  average  of  50  percent  of  t  e 

standards.  Its  performance  units  totaled  600  (1,200  x  0.50).  In 
case  "b, "  performance  is  higher,  at  960  units  (1,200  x  0.80), 
reflecting  a  higher  level  of  quality.  Case  "c"  shows  the  lowest 
number  of  performance  units,  560  (1,400  x  0.40),  because  of  a 

significantly  lower  quality  level.  These  three  cases  illustrate 
that  accurate  performance  measurement  must  standardize  for  quality 
differences,  and  that  organizations  could  perform  poorly  if  they  do 
not  embrace  such  comprehensive  measures. 
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Table  4-2 
Performance  Units 


Case 

Output 

(0) 

Quality  index 
(QI) 

Performance 
unit  (OxQI) 

a 

1,200 

0.50 

600 

b 

1,200 

0.80 

960 

c 

1,400 

0.40 

560 

Performance  Unit  Cost 

The  starting  point  for  defining  performance  unit  cost  is  to 
establish  the  relationship  between  total  costs,  on  the  one  hand, 
and  output  and  quality,  on  the  other.  Economists  call  such  a 
relationship  a  "total  cost  function";  its  properties  can  be  derived 
by  either  regression  analysis  or  engineering  value  analysis. 
Skilled  professionals  would  be  needed  to  undertake .  such  technical 
analyses.6  Equation  4-2  represents  a  total  cost  function. 

Total  Cos t=f (output,  quality)  [Eq.4-2] 


It  indicates  that  costs  vary  with  both  output  and  quality,  and  that 
such  a  relationship  could  be  either  linear  or  nonlinear,  depending 
upon  the  particular  circumstances. 


Performance  unit  cost  (or  PUC)  simply  expresses  the  total  cost 
relative  to  performance  units.  Dividing  both  sides  of  Equation  42 
by  the  performance  units  (output  x  quality) ,  we  can  calculate  PU  , 
as  Equation  4-3  shows: 

PUC=f (output,  quality) /performance  units  [Eq.4-3] 

Note  that  PUC  is  not  a  constant  but  varies  with  output  and 
quality.  Many  budgeting  systems  assume  that  PUC  is  constant.  Such 
an  assumption,  however,  could  lead  to  incorrect  managerial 
decisions,  especially  when  related  to  consolidations  and 


6  See  Jan  Kmenta,  Elements  of  Econometrics,  New  York: 
Macmillan  Publishing  Co.,  Inc.,  1971,  for  a  good  treatment  of 
regression  analysis.  Also,  see  David  K.  Burt,  et  al.,  Zero  Base 
Pricing:  Achieving  World  Class  Competitiveness  Through  Reduced 
All-in  Costs,  Chicago:  Probus  Publishing  Company,  1990,  for  a 
thorough  explanation  of  engineering  value  analysis. 
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realignments.  Organizations  should  use  a  constant  PUC  only  as  a 

very  crude  first  approximation. 

-sra  HU 

performance  units  become  large  (such  as  at  point  a  . 

The  PUC  curve  can  be  used  for  various  managerial  Proses .  For 

•  • +-  r>oni  ri  Vip  used  to  summarize  organizational  progress  fo 

instance,  it  could  be  used  uo  sunuio  .=  initially  operates  at 

making  improvements.  Suppose  ®  ®  >,iah  puc  That  organization 

point  -b,-  signifying  a  rela“Xfiy„  initiatives  to  improve 

might  pursue  cost  reduction  and/or  '“at  level  of 

its  performance.  If  it  simp  y  noint  "c")  then  its  PUC  would 

performance  units  (such  as  p  focuses'  on  quality  enhancement 

decline  from  PUC <*>>  ^  pu^>  puc  als0  would  decline,  but  less 

(such  as  at  point  a), (  then  ^  Either  enhancing  quality  or 

Cueing"11 coys'ts  may  be  appropriate,  depending  upon  customer 

feedback. 

The  PUC  curve  could  also  be  used  ^o  target  the^primSy 

initiatives.  If  customers  indicate  that  c  co  cost 

to  "a" )  . 

Final lv  the  PUC  curve  could  be  used  to  develop  budgets.  As  an 

example,  the ^pujfaj^upta) ; ^or^hfbudge/to  °^ate  at"  UP(b) 
UP  (a)  would  be  PUC  (a)  x  '  nr  /  )x  UP(b)  after  cost-reduction 

performance  units  would  be  PUC(c)x  ufw 

efforts. 


7  An  example  of  a  linear  cost  function  might  be  as  follows: 

i  ,  -u  (outnut  x  Quality  Index)  .  In  this  situation. 

Total  cost  =  a  +  b  Output  x^oua^  ,y  constant^  puc  =  a/  (output  x 

marginal  cost  is  S11  first  term  reflecting  average  fixed 

SStsIwSorla^y  InviSeS  with  output  and  quality)  and  the  last 

term  ref lecting^ihe  constant  average  variable  or  marginal  costs. 
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Unit  Performance 


FIGURE  4-2.  EFFECT  OF  LINEAR  COST  FUNCTION  ON  PUC 


WIND-TUNNEL  CASE 

Using  our  definition  of  processes,  the  Ames  Research  Center 
(ARC)  wind-tunnel  operations  could  be  classified  as  a  single 
process  for  measuring  performance  and  making  improvements.  It  is 
our  understanding  that  ARC  operates  three  different  types  of  wind 
tunnels:  a  7-by-10-foot  high-speed  tunnel;  a  40-by-80-foot 

moderate-speed  tunnel;  and  an  80-by-120  foot  low-speed  tunnel. 
Each  of  those  tunnels  supports  different  customers.  Also,  ARC 
schedules  wind-tunnel  usage  in  order  to  minimize  power, 
maintenance,  and  repair  costs.  Finally,  the  ARC  wind  tunnels 
further  support  NASA's  own  operations  (e.g.,  in  flight  orbiter 
preparations)  and  industry  (e.g.,  in  helicopter  research). 

In  illustrating  our  performance  measurement  system  with  the 
ARC  wind-tunnel  operation  as  a  single  process,  we  need  to  first 
define  and  formulate  the  elements  of  performance  units  for  the 
single  wind-tunnel  process:  cost,  output,  and  quality.  Then,  we 
need  to  formulate  the  total  cost  and  PUC  curve,  as  well  as 

illustrate  their  application  to  management. 

Cost 

The  total  costs  (TC)  for  ARC'S  wind-tunnel  process  may  be 
broken  down  into  two  components:  operating  costs  and  depreciation 

charges.  Operating  costs  (OC)  consist  of  power,  maintenance, 

repairs,  and  other  variable  costs,  as  well  as  some  administrative 
and  fixed  costs  that  are  associated  with  the  wind-tunnel  operation. 
Depreciation  charges  (DC)  reflect  the  amortization  of  expenditures 
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for  the  wind-tunnel  facilities,  equipment,  and  other  capital  items. 
Equation  4-4  shows  the  straightforward  calculation  of  TC,  in 
constant  dollars.  It  reflects  the  _  real  resources  necessary  to 
provide  wind-tunnel  services  at  certain  levels  of  quality. 


TC=OC+DC 


[Eq.  4-4] 


Output 

The  wind-tunnel  outputs  consist  of  low-,  medium-,  and  high¬ 
speed  wind-tunnel  services.  Each  of  those  service  outputs  could  e 
measured  in  terms  of  usage  hours— low- (1) ,  medium- (m) ,  or  high  (h) 
speed  hours.  In  order  to  combine  these  outputs  into  a  composite,  we 
need  to  weight  them  to  reflect  their  relative  contributions  to  the 
?otal  ARC  wind-tunnel  service.  We  believe  that  relative  cost 
weicrhts  would  best  serve  this  purpose.  Equation  4-5  shows  the 
calculation^1  of  the  output  index  (01)  for  the  ARC  wind  tunnel 

process . 

01  =  w ( 1 )  x  1  +w(2)  x  m  +w(3)  x  h  [Eq.  4-5] 

Quality 

The  wind-tunnel  services  are  very  likely  to  vary  in  quality  in 
a  number  of  respects.  We  have  chosen  three  quality  dimensions  to 
illustrate  this  point:  suitability,  usability,  and  reliability 
Suitability  relates  to  the  adequacy  of  the  ARC  wind  tunics  offered 
for  meeting  the  testing  requirements  of  its  customers.  Usability 
refers  to  whether  the  wind-tunnel  service  schedules  are  reasonable 
for  meeting  the  needs  of  both  NASA  and  industry  customers. 
Reliability  indicates  whether  the  wind  tunnel  operation  performs 
well  and  without  interruption.  Each  of  these  quality  dimensions  i 
expressed  as  a  percentage  of  actual  performance  as  compared  to  tne 
standard  measures.  We  then  need  to  combine  these  quality  measures 
into  an  overall  quality  index  (QI)  using  Equation  4-6. 

QI=i ( 1 )  x  (%suit. )  +i  (2)  x  (%use. )  +i(3)  x  (%  rel.)  [Eq.4-6] 


The  quality  factor  weights  in  the  index  i(l),  i(2),  and  id) 

should  reflect  the  relative  importance  that  industry  and  NASA 
customers  attach  to  them.  A  customer  survey  could  use  a  l-to-7 
scoring  system  (see  Section  3)  to  determine  customer  standards  and 
satisfaction  for  meeting  those  standards. 
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Performance  Unit  Cost 

To  obtain  performance  unit  cost  for  the  ARC  wind-tunnel 
process,  we  need  to  follow  three  sequential  steps: 

(a)  Define  unit  of  performance  (UP) using  the  formula  UP  =  01 
x  QI,  where  01  is  computed  using  Equation  4-5  and  QI  using  Equation 
4-6. 


(b)  Establish  the  relationship  between  total  cost  and  the 
quality  and  output  indices.  Both  linear  and  non-linear  relation¬ 
ships  should  be  considered;  they  can  be  established  by  using  either 
regression  analysis  or  engineering  value  analysis. 

(c)  Calculate  performance  unit  cost  by  dividing  the  UP 
measure  from  (a)  into  the  cost  equation  established  in  (b) . 

Management  Uses 

With  the  quality  index,  ARC  can  then  determine  where  it  needs 
to  make  improvements  by  expressing  the  quality  index  (Equation  4-6) 
in  change  or  complete  difference  form  as  in  Equation  4-7: 

d(QI) =i  (1)  x  d(%suit) gap  +i(2)  x  d(%use)gap  +i(3)  x  d(%rel)gap 

[Eq. 4-7] 

The  gap  for  each  quality  factor  indicates  the  remaining 
difference  between  the  customer's  standard  and  actual  satisfaction 

level.  For  example,  if  a  customer  indicates  that  60  percent  of  the 

standard  level  is  met  for  suitability,  the  remaining  actual 
standard  gap  is  40  percent.  If  the  weight  for  suitability  is  0.60, 
then  improvements  in  suitability  can  contribute  a  maximum  of  24 
percent  to  overall  wind-tunnel  process  quality  (40  percent  gap  x 
0.60  weight  in  quality  index).  ARC  could  also  determine  the 
relative  contributions  of  the  other  quality  factors  to  the  overall 
potential  improvement  in  ARC  wind-tunnel  process  quality.  Armed 
with  this  information,  ARC  then  could  prioritize  its  improvement 

efforts  according  to  their  relative  contributions  to  changes  in 
overall  quality. 

ARC  could  use  this  same  performance  measurement  system  for 
other  managerial  purposes.  Through  the  customer  survey,  it  could 
determine  the  importance  of  wind-tunnel  expenditures  to  the  users 
and  then  compare  that  result  to  potential  quality  initiatives..  If 

the  customers  consider  expenditures  more  important  than  quality, 
ARC  should  focus  its  process  improvements  on  reducing  costs. 
Finally,  ARC'S  overall  progress  on  wind-tunnel  process  improvements 
and  budgeting  requirements  could  be  derived  from  the  performance 
unit  cost  curves  shown  in  Figure  4-2. 


23 


